{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "58b30d1f-28e7-4e27-b256-f7c7bbcfbf4b",
   "metadata": {
    "id": "58b30d1f-28e7-4e27-b256-f7c7bbcfbf4b"
   },
   "source": [
    "## InstructOR - A multitask custom embedding model for task based applications, made easier with LanceDB\n",
    "![instruct](https://github.com/lancedb/vectordb-recipes/blob/main/examples/instruct-multitask/embeddings11.png?raw=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2506f81b-cf52-41f2-9f0d-6b20836b15b6",
   "metadata": {
    "id": "2506f81b-cf52-41f2-9f0d-6b20836b15b6"
   },
   "source": [
    "### Installing all dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d8c13226-8a68-485a-86ba-6243eb1ed26e",
   "metadata": {
    "id": "d8c13226-8a68-485a-86ba-6243eb1ed26e"
   },
   "outputs": [],
   "source": [
    "!pip install lancedb -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9bb0ec2f-9953-427b-b358-c9af1a78a61e",
   "metadata": {
    "id": "9bb0ec2f-9953-427b-b358-c9af1a78a61e"
   },
   "outputs": [],
   "source": [
    "!pip install InstructorEmbedding sentence-transformers==2.2.2 torch pandas -q"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4ce6fbd-75fb-4663-b68f-716295ae3720",
   "metadata": {
    "id": "b4ce6fbd-75fb-4663-b68f-716295ae3720"
   },
   "source": [
    "If you want to calculate customized embeddings for specific sentences, you may follow the unified template to write instructions:\n",
    "\n",
    "\"Represent the [**domain**] [**text_type**] for [**task_objective**]:\"\n",
    "\n",
    "Here are some examples:\n",
    "\n",
    "- \"Represent the **Science** **sentence**:\"\n",
    "- \"Represent the **Financial** **statement**:\"\n",
    "- \"Represent the **Wikipedia** **document** for **retrieval**:\"\n",
    "- \"Represent the **Wikipedia** **question** for **retrieving supporting documents**:\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7be2cd8c-710f-4eca-b93e-2b5f5876198a",
   "metadata": {
    "id": "7be2cd8c-710f-4eca-b93e-2b5f5876198a"
   },
   "source": [
    "### Importing neccessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "63f36f93-cf85-4548-a66c-5e4c8b8f9401",
   "metadata": {
    "id": "63f36f93-cf85-4548-a66c-5e4c8b8f9401"
   },
   "outputs": [],
   "source": [
    "import lancedb\n",
    "from lancedb.pydantic import LanceModel, Vector\n",
    "from lancedb.embeddings import get_registry\n",
    "from lancedb.embeddings import InstructorEmbeddingFunction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3258997-580c-4231-b578-65db806425ef",
   "metadata": {
    "id": "c3258997-580c-4231-b578-65db806425ef"
   },
   "source": [
    "### Calling the embedding model from LanceDB embedding's API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9aaa6835-fa7c-4197-b86b-e25efd7a7eb3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 667,
     "referenced_widgets": [
      "85154f1dc1274691a910972c70b6fc77",
      "146303e934ca4a12be1ec8c86822fc45",
      "e8e587d6d82c4806b1923187253d968a",
      "1acefe91f495444a85f6d3621819d568",
      "b1e07fb56c2d46ee8bee500953dd25f9",
      "eaff4437e0014a69a7f7660703c82ce4",
      "026fea539acd4574a8ea1c28f45f3212",
      "206cc33a0ffd4c1588a0b9fad8555ac2",
      "865144e6d98941be9b0b23afbbeee301",
      "9d5afef396d84452b789fd9fd1521b05",
      "cb5ea370b245428093848d225f3cb213",
      "8cd7cba5a4c44afaa64b02526d4ff6fd",
      "f68e902f2bb243e5ab4bd71655750881",
      "d88942d4fc984a00ae36cb3c786f504d",
      "a36fa49b9bcb424a992fd733a543bc82",
      "f4ba6b324995497cac764f105ad8200a",
      "e637c489831c41a7b1c6adfe6dc3b0ef",
      "6fa25acd8bb14a08a8baf9c911663bb5",
      "2e9e3e6d0dbc471899d21a5fdb4d36b4",
      "fd54d9c191fa41d4847b2de48839b633",
      "0ea4ddfc43134c72ac0e154aa94b0a9b",
      "d5366053462a4798bebca57223266c3f",
      "46f288860745416382c9c887d3d6904c",
      "2fbbb5b9656c42f194e5715a56de0ad8",
      "4e2e240ffbf74adc93265d476d84aa3a",
      "40edc13e75284e7da77167553325f8c5",
      "d068f9b0357544a2adea1612c5e7324f",
      "b68def2ec8d843cba2e3efd275e6df07",
      "1a1f0fd9040641c3b0201ced7ff29601",
      "21c2a1de412c4cd3bb8e271c12d11414",
      "613fe4db800146d99351c0b3772c67f0",
      "acb90eeace364a7f94fed50b528734a2",
      "f856e6239e90400e811ffb0cca95e389",
      "5ddf1e47f6d94448a1f9bfff6fec1ff0",
      "4bd748695953457a8c71d92c3c935cd4",
      "d089bee6435f4ad290b9c3d6f85eb363",
      "bd350e75cbe94f2ab3324557d027c1f4",
      "e564e1ac5c214218b60137bd9bcaa43c",
      "86088f41815640449165c86f5cae9a6d",
      "3342aeb2dd834cc1a591c886a378d356",
      "ce58257c41e24f21841af78ba0c16e6f",
      "65f0e32d810042f48b2ef99b4dc07c34",
      "6ed2aaad552f431e9b6e71953f51d35f",
      "4f0563e1c6294ccd85f8f8a83e59a014",
      "9904be446b564ca8aa7989c97c0abbab",
      "d6b5e65458d24422bc870f086cb1fa1a",
      "eef42b8fc13148bda5d351031a3eb4da",
      "fe91072ca7cc419aa16794fa8dc64e10",
      "3a27e26bed54439d8d70913eea725612",
      "016cea53eec044eda739d54b73fe0f06",
      "7ff56eb034c24cbaa044e16d815263e0",
      "c6bf2108a71049fb8363bcb2438a1289",
      "f25f520520f0416f9ceecf407cd6106c",
      "2ba4eb134ad84d2c8d197866c7471abf",
      "151c243ecbbe4862a83e20f8a18d01fe",
      "914497fe6dee4460afa37574ffb89c58",
      "cc1a242592fd460ca98473e5dd9649af",
      "5a32c7c7da73441097a8f6f04027c728",
      "899a8bc6bce247ba8940771aca170e35",
      "c3d673ecd2854f70bb30e4f57ff15c50",
      "3091ace1e6754f06aa346514b22a2a6c",
      "f9b3d9ed32694ebeb48f602a5be994fb",
      "a14f301b05ff4112818ca22d12db0c6a",
      "2a8492da2fc9400f9a2e9c0665c32efb",
      "c36a3c966d1d4fea95af1f106208be56",
      "d5af404072b64b17b88a524719f1ae20",
      "b75a97fc8a43488d8b2dc194f2ad3036",
      "1e6017c9d2c64b09a2f83c81507c00be",
      "b773e91d6bea4720889ff0460a5c1e01",
      "b1c6170862774cbaba577c5fc625bd7f",
      "c99b7e8dd23c4269b5a637e7443a40df",
      "e681c6a48f074f80aa1d7f521513d324",
      "15ea8e054acb4846bb79e9088d335b5b",
      "36e274a681934de09ed70635bbbdf5b4",
      "c94e7bd4ca514af5b7b35c892795c8d3",
      "a9f1318538ba40f0a8eb03ddea8e2491",
      "a50c4ffb010743868242531ef8cdc608",
      "c4a144c3512449308107c5202429c760",
      "9b548ea17e19414bae3fddced420ae24",
      "b4f5af420dc04d3691c0b22660e5ccf9",
      "372778777e86407484a7802621b9de6f",
      "d16d0f06fdb343d9b9f04bc29770eb5c",
      "34682ca7085c44dab1570fc1ed5b823a",
      "26ba46bb0da14051a3703e9947078d9a",
      "a58bfd15eb6447549f4b8c01db84f1f3",
      "f35b42ef3cda4a95b597ac328fc1c910",
      "36983e2a9afb4dceb0ed71c6af61e592",
      "fc62826026214d9997d9627130e2f227",
      "d68346e83dc342759c82cebac1cbdb22",
      "834cbd0c37524eedbb7068fcee3e3189",
      "ccbe576d2d6d4c2dab72ed06d47af3b5",
      "9063963e14fb499cbef4cd56cc604894",
      "b1e41b746ed54e76948d7412602a8d60",
      "61cb56a358c84d2aae433b2d73bc0831",
      "f3bb49a5c1b24bd6b229fd6f1718c90d",
      "360bc995f7964f60baa6813efdd6168e",
      "9e1c3cbbf4e044bf93a847aafe0a756f",
      "3f2d4f47a2a54ead8dfae18ae740de21",
      "b5df3155c1a145639dcd3544fd1cee4a",
      "a5fab68c0fa940eaa66e5c87b18bd6fb",
      "9e3f5cb3c95844e4acdc96ea90debfbd",
      "247b0ae8b8bc431f9229ae46930953fa",
      "add6d13dc27b4170bcab6d0ee1ce053c",
      "074420d5af0049e18245eb0004ebae10",
      "1fa4769b505b4fe486e4d87bc372b93f",
      "5a43dc86a146415285c2121dd03706dd",
      "0f0cc64532324471a740ac27ae31089c",
      "b895a002160b4711b884031fc2ede9f1",
      "198ec9c3a25c47e49f5d76dc6a639c6e",
      "119b5746e7a44520a0c4b98d62113aff",
      "dedd82e8bc774b52a30019d0e8506b9c",
      "18898b7d565f4a459f45ebcdc92ef2c1",
      "a21bee33607141a08fcbdb39386e7d9f",
      "c95fafa6bd164aff8661523752090c7a",
      "000b3baf1550439f83b13d0e264b1d27",
      "d247becea58949f693bd9a7dd75f6f62",
      "33455117e6b9401dabbe71a2437d3e98",
      "eb564047d3d94fed9f596e5f9dada819",
      "6f0a74a9d6ad46fd92c503f37c8dd4a9",
      "8b56e9b5133a4357a1a7b971f6fe5d2d",
      "ec9c40e97a36430d8e30e8e17e600c0a",
      "134fa6bcb1664543863846d2f59b9343",
      "70376de4fc784b6fb084cfc9f6e51086",
      "d41d291b34974317ab8c1eab6ddec769",
      "4080df235524446885af2d446fb55856",
      "9e0de297fd884b269dc0d32aa64d27f2",
      "dd011645d83c401db9e32d4c7e3b5370",
      "7ba5c5db9eea43d3b896b7b62907385e",
      "64df69b09bd2419c8c5efccef6e237af",
      "10c201f1c876424587c9b39a03c2e0ca",
      "96fa250ab07643949333a97489926667",
      "74b1310bde5b43d5b32bd62a9a360ced",
      "930b3a33f11549e68aba4eddef81dbfd",
      "44c3df6eba2e430b981390e73515412b",
      "15763d7dc6cc4685a14c1a2362d4d694",
      "6e09649d1ead40ce8b63712c79d64294",
      "25cda513d18f49baaf57fa886d7768e8",
      "866949171d8e4d7bbbd2527c533826c2",
      "58f53c926d214bc7b73ecee67571bccd",
      "01d2bb8006c74986ab97903cef428f93",
      "70091ac5714d4773a7563bb9690f7a52",
      "0783bde86e7b41f5985bd38e44461307",
      "4b0336be407e45e983b4fb0d01bc3136",
      "f101500091fb4f319835353a1f1fd7b0",
      "9cfe53d29ff64dc4a7277fef8d8f3576",
      "bf0ca8f97ccd460cb4d3b41e5a344593",
      "47d6503de0a24a048f029d478b36016a",
      "f690d5440c9b471a9f64bdd16e943f6f",
      "d8c4aa5d0a8e4b38a0186783d3585b48",
      "b5d40fbda8214b28a2d8449b2b6340d1",
      "a40187d3641d402fa88810aee0345e4a",
      "8ffccd9914f4439bacdee768a4f8e4e1",
      "be6e34592b2b471d8fc8a214ae69f537",
      "9ab52a6b293f4502acff0e22542d968d"
     ]
    },
    "id": "9aaa6835-fa7c-4197-b86b-e25efd7a7eb3",
    "outputId": "74a6cc7a-072a-4a6c-eeaa-b1d3533d7739"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "85154f1dc1274691a910972c70b6fc77",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       ".gitattributes:   0%|          | 0.00/1.48k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8cd7cba5a4c44afaa64b02526d4ff6fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "1_Pooling/config.json:   0%|          | 0.00/270 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46f288860745416382c9c887d3d6904c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "2_Dense/config.json:   0%|          | 0.00/115 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5ddf1e47f6d94448a1f9bfff6fec1ff0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/2.36M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9904be446b564ca8aa7989c97c0abbab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "README.md:   0%|          | 0.00/66.2k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "914497fe6dee4460afa37574ffb89c58",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/1.55k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b75a97fc8a43488d8b2dc194f2ad3036",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config_sentence_transformers.json:   0%|          | 0.00/122 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4a144c3512449308107c5202429c760",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/439M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d68346e83dc342759c82cebac1cbdb22",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "sentence_bert_config.json:   0%|          | 0.00/53.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a5fab68c0fa940eaa66e5c87b18bd6fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/2.20k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dedd82e8bc774b52a30019d0e8506b9c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "spiece.model:   0%|          | 0.00/792k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "134fa6bcb1664543863846d2f59b9343",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/2.42M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "930b3a33f11549e68aba4eddef81dbfd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/2.43k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f101500091fb4f319835353a1f1fd7b0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "modules.json:   0%|          | 0.00/461 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load INSTRUCTOR_Transformer\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()\n",
      "  return self.fget.__get__(instance, owner)()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "max_seq_length  512\n"
     ]
    }
   ],
   "source": [
    "instructor = (\n",
    "    get_registry()\n",
    "    .get(\"instructor\")\n",
    "    .create(\n",
    "        source_instruction=\"represent the document for retreival\",\n",
    "        query_instruction=\"represent the document for most similar definition\",\n",
    "    )\n",
    ")\n",
    "\n",
    "\n",
    "class Schema(LanceModel):\n",
    "    vector: Vector(instructor.ndims()) = instructor.VectorField()\n",
    "    text: str = instructor.SourceField()\n",
    "\n",
    "\n",
    "# Creating LanceDB table\n",
    "db = lancedb.connect(\"~/.lancedb\")\n",
    "tbl = db.create_table(\"intruct-multitask\", schema=Schema, mode=\"overwrite\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6912dbd-66b6-4a8f-a1aa-78c78536dec0",
   "metadata": {
    "id": "c6912dbd-66b6-4a8f-a1aa-78c78536dec0"
   },
   "source": [
    "### Adding Data to the Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a4959ee0-2e84-470d-aa95-cec459e2bb1b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "2704bfd8c7404c89bdec3e67af674431",
      "977c6e2ac1404ffe880cea76616524ff",
      "4ddb89482d824b83821a56a2def3d645",
      "bd540b1ce9bb471cafabf14f0c3cef12",
      "c36fa236492940fe89036d23df67585f",
      "5239acdd75504fc4b2f4aacd994aa360",
      "f760e49da27c4eeda776a6f64e3a065a",
      "2e84617630a34b378705cf80fd08a316",
      "73cce42a9a6542e0a1d63bbe9ff7f2b4",
      "eca5eb7b237a453fbb6bc7e36c110cf2",
      "1ff763c769d045a9842be59f7cf0aaa6"
     ]
    },
    "id": "a4959ee0-2e84-470d-aa95-cec459e2bb1b",
    "outputId": "cc1205de-8e3e-4daa-bec0-f2bca1676888"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2704bfd8c7404c89bdec3e67af674431",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_f1 = [\n",
    "    {\n",
    "        \"text\": \"Aspirin is a widely-used over-the-counter medication known for its anti-inflammatory and analgesic properties. It is commonly used to relieve pain, reduce fever, and alleviate minor aches and pains.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Amoxicillin is an antibiotic medication commonly prescribed to treat various bacterial infections, such as respiratory, ear, throat, and urinary tract infections. It belongs to the penicillin class of antibiotics and works by inhibiting bacterial cell wall synthesis.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Atorvastatin is a lipid-lowering medication used to manage high cholesterol levels and reduce the risk of cardiovascular events. It belongs to the statin class of drugs and works by inhibiting an enzyme involved in cholesterol production in the liver.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"The Theory of Relativity is a fundamental physics theory developed by Albert Einstein, consisting of the special theory of relativity and the general theory of relativity. It revolutionized our understanding of space, time, and gravity.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Photosynthesis is a vital biological process by which green plants, algae, and some bacteria convert light energy into chemical energy in the form of glucose, using carbon dioxide and water.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"The Big Bang Theory is the prevailing cosmological model that describes the origin of the universe. It suggests that the universe began as a singularity and has been expanding for billions of years.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Compound Interest is the addition of interest to the principal sum of a loan or investment, resulting in the interest on interest effect over time.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Stock Market is a financial marketplace where buyers and sellers trade ownership in companies, typically in the form of stocks or shares.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Inflation is the rate at which the general level of prices for goods and services is rising and subsequently purchasing power is falling.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Diversification is an investment strategy that involves spreading your investments across different asset classes to reduce risk.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Liquidity refers to how easily an asset can be converted into cash without a significant loss of value. It's a key consideration in financial management.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"401(k) is a retirement savings plan offered by employers, allowing employees to save and invest a portion of their paycheck before taxes.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Ballet is a classical dance form that originated in the Italian Renaissance courts of the 15th century and later developed into a highly technical art.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Rock and Roll is a genre of popular music that originated and evolved in the United States during the late 1940s and early 1950s, characterized by a strong rhythm and amplified instruments.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Cuisine is a style or method of cooking, especially as characteristic of a particular country, region, or establishment.\"\n",
    "    },\n",
    "    {\"text\": \"Renaissance was a cultural, artistic, and intellectual movement that\"},\n",
    "    {\n",
    "        \"text\": \"Neutrino is subatomic particles with very little mass and no electric charge. They are produced in various nuclear reactions, including those in the Sun, and play a significant role in astrophysics and particle physics.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Higgs Boson is a subatomic particle that gives mass to other elementary particles. Its discovery was a significant achievement in particle physics.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Quantum Entanglement is a quantum physics phenomenon where two or more particles become connected in such a way that the state of one particle is dependent on the state of the other(s), even when they are separated by large distances.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Genome Sequencing is the process of determining the complete DNA sequence of an organism's genome. It has numerous applications in genetics, biology, and medicine.\"\n",
    "    },\n",
    "]\n",
    "\n",
    "tbl.add(data_f1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81a5eca6-cdb3-46e6-b2d4-5d9a55b65783",
   "metadata": {
    "id": "81a5eca6-cdb3-46e6-b2d4-5d9a55b65783"
   },
   "source": [
    "## First use case - Semantic Search with LanceDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6111c40b-588f-401d-af8a-1c729e51d0b8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "b097b660d02141e9b6b88f0d4f393053",
      "742919deb70140278b81e736bab6d90b",
      "acb2796b71c94a85868787d9445ab0a5",
      "f75396c58d7f44aead5a0d292fc108b4",
      "91cc8ae714144f64a6f4ad95fe6379ab",
      "02cf51a22807451f9f4439d48da000d0",
      "c7f019db913b4dd2b35b4cb8943e1a23",
      "c35837e1f9314cc9a782158578f3933c",
      "35af160009fc4a76a4652e53d947f5e8",
      "f96721b296f14f71a09547e661ae9bbc",
      "8666d8447a1a415596829548082817b7"
     ]
    },
    "id": "6111c40b-588f-401d-af8a-1c729e51d0b8",
    "outputId": "553fb7e3-b757-4be2-ea7d-be93ccf2cd6e"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b097b660d02141e9b6b88f0d4f393053",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# LanceDB supports full text search, so there is no need of embedding the Query\n",
    "query = \"amoxicillin\"\n",
    "result = tbl.search(query).limit(1).to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "72fedb02-c63a-4d25-86bc-1c3c4f7ca8e9",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "72fedb02-c63a-4d25-86bc-1c3c4f7ca8e9",
    "outputId": "49dc199d-7123-46c9-bfde-b0d12612dd1a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              vector  \\\n",
      "0  [-0.024510663, 0.0005563084, 0.028840268, 0.08...   \n",
      "\n",
      "                                                text  _distance  \n",
      "0  Amoxicillin is an antibiotic medication common...   0.163671  \n"
     ]
    }
   ],
   "source": [
    "# printing the output\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ea9ebd-5301-43c2-8149-9506f10bfd3b",
   "metadata": {
    "id": "98ea9ebd-5301-43c2-8149-9506f10bfd3b"
   },
   "source": [
    "## Same Input Data with Different Instruction Pair"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "54481449-cc7b-4b62-ae40-9bd5e91eb0af",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "54481449-cc7b-4b62-ae40-9bd5e91eb0af",
    "outputId": "99f5976d-f7ba-44f3-9297-5894705d7358"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load INSTRUCTOR_Transformer\n",
      "max_seq_length  512\n"
     ]
    }
   ],
   "source": [
    "instructor = (\n",
    "    get_registry()\n",
    "    .get(\"instructor\")\n",
    "    .create(\n",
    "        source_instruction=\"represent the captions\",\n",
    "        query_instruction=\"represent the captions for retrieving duplicate captions\",\n",
    "    )\n",
    ")\n",
    "\n",
    "\n",
    "class Schema(LanceModel):\n",
    "    vector: Vector(instructor.ndims()) = instructor.VectorField()\n",
    "    text: str = instructor.SourceField()\n",
    "\n",
    "\n",
    "db = lancedb.connect(\"~/.lancedb\")\n",
    "tbl = db.create_table(\"intruct-multitask\", schema=Schema, mode=\"overwrite\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e1c5fdab-5a0b-47ce-826e-2fb899570431",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "571e47c352e3431f8bf2007f474663bd",
      "f8a922caa7f74fb4bf086bc159bcb4c0",
      "339de2865fe941e482c9b8cf01f7a552",
      "353151490030427eb35c32b69fdc45b2",
      "fe89b2fe5c27452c9ed9bb20db6edd0b",
      "153fdc7143994a2da0c2198a88c417e9",
      "707b99fe1dde4beaadb38e9244922ea4",
      "903a8d88329547189b2feae373e76072",
      "b12e5193e8b840ed97e8014e334f5a2b",
      "d3bfa73a4f92480c91a881c0a8528928",
      "d961cf3e80f743e1ae24bf30f15b6e66"
     ]
    },
    "id": "e1c5fdab-5a0b-47ce-826e-2fb899570431",
    "outputId": "a84fdafe-a778-4010-fef1-c11bca7c2951"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "571e47c352e3431f8bf2007f474663bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_f2 = [\n",
    "    {\n",
    "        \"text\": \"Aspirin is a widely-used over-the-counter medication known for its anti-inflammatory and analgesic properties. It is commonly used to relieve pain, reduce fever, and alleviate minor aches and pains.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Amoxicillin is an antibiotic medication commonly prescribed to treat various bacterial infections, such as respiratory, ear, throat, and urinary tract infections. It belongs to the penicillin class of antibiotics and works by inhibiting bacterial cell wall synthesis.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Atorvastatin is a lipid-lowering medication used to manage high cholesterol levels and reduce the risk of cardiovascular events. It belongs to the statin class of drugs and works by inhibiting an enzyme involved in cholesterol production in the liver.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"The Theory of Relativity is a fundamental physics theory developed by Albert Einstein, consisting of the special theory of relativity and the general theory of relativity. It revolutionized our understanding of space, time, and gravity.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Photosynthesis is a vital biological process by which green plants, algae, and some bacteria convert light energy into chemical energy in the form of glucose, using carbon dioxide and water.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"The Big Bang Theory is the prevailing cosmological model that describes the origin of the universe. It suggests that the universe began as a singularity and has been expanding for billions of years.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Compound Interest is the addition of interest to the principal sum of a loan or investment, resulting in the interest on interest effect over time.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Stock Market is a financial marketplace where buyers and sellers trade ownership in companies, typically in the form of stocks or shares.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Inflation is the rate at which the general level of prices for goods and services is rising and subsequently purchasing power is falling.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Diversification is an investment strategy that involves spreading your investments across different asset classes to reduce risk.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Liquidity refers to how easily an asset can be converted into cash without a significant loss of value. It's a key consideration in financial management.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"401(k) is a retirement savings plan offered by employers, allowing employees to save and invest a portion of their paycheck before taxes.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Ballet is a classical dance form that originated in the Italian Renaissance courts of the 15th century and later developed into a highly technical art.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Rock and Roll is a genre of popular music that originated and evolved in the United States during the late 1940s and early 1950s, characterized by a strong rhythm and amplified instruments.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Cuisine is a style or method of cooking, especially as characteristic of a particular country, region, or establishment.\"\n",
    "    },\n",
    "    {\"text\": \"Renaissance was a cultural, artistic, and intellectual movement that\"},\n",
    "    {\n",
    "        \"text\": \"Neutrino is subatomic particles with very little mass and no electric charge. They are produced in various nuclear reactions, including those in the Sun, and play a significant role in astrophysics and particle physics.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Higgs Boson is a subatomic particle that gives mass to other elementary particles. Its discovery was a significant achievement in particle physics.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Quantum Entanglement is a quantum physics phenomenon where two or more particles become connected in such a way that the state of one particle is dependent on the state of the other(s), even when they are separated by large distances.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"Genome Sequencing is the process of determining the complete DNA sequence of an organism's genome. It has numerous applications in genetics, biology, and medicine.\"\n",
    "    },\n",
    "]\n",
    "\n",
    "tbl.add(data_f2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b6bcee5e-192f-4eeb-9ad0-c4c6e60f936d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "a93af2daec7f48038d5387563427e313",
      "9c1d8032530e47559959b3534e115d56",
      "0bac01cca80447ca9d57ccc5f99715c7",
      "d6b819549951460ea20b589216ac1684",
      "03ff96d46cad47f09d7c5d74c2f74e98",
      "6d67683fbfcd423fa6fc6b070d42e14f",
      "69bab3e622b240ce92a6334b6d4ace90",
      "fd690bb88ff94e75a168236c12508559",
      "d4be3f9d4ba641628a1f09834322ac38",
      "a48cb12a70704abaaddd8d7a2d1afe82",
      "c874df22aa6b42ffbac199422d3cfd90"
     ]
    },
    "id": "b6bcee5e-192f-4eeb-9ad0-c4c6e60f936d",
    "outputId": "d7463657-5d59-4333-cc4f-0e28d90b8d9b"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a93af2daec7f48038d5387563427e313",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# same query, but for the differently embed data\n",
    "query = \"amoxicillin\"\n",
    "result = tbl.search(query).limit(1).to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "895ddc86-ecec-4403-acc8-0b51cb04178c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "895ddc86-ecec-4403-acc8-0b51cb04178c",
    "outputId": "1b0256dc-9f07-4a09-e25d-93ad11b3d042"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              vector  \\\n",
      "0  [-0.02448329, 0.00093284156, 0.033273738, 0.07...   \n",
      "\n",
      "                                                text  _distance  \n",
      "0  Amoxicillin is an antibiotic medication common...    0.18135  \n"
     ]
    }
   ],
   "source": [
    "# showing the result\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ac26b83-a13f-4fc7-baaf-ff6817baab3d",
   "metadata": {
    "id": "5ac26b83-a13f-4fc7-baaf-ff6817baab3d"
   },
   "source": [
    "### We can see that the **_distance** value for different instructions are different, this clearly indicates that the instructions have some effect on the embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae29780a-7bc7-4baf-94e3-561835e18b8f",
   "metadata": {
    "id": "ae29780a-7bc7-4baf-94e3-561835e18b8f"
   },
   "source": [
    "## Second use case - **Question Answering** with LanceDB"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a08ec791-bd1f-47d9-96f7-70bdbaf24769",
   "metadata": {
    "id": "a08ec791-bd1f-47d9-96f7-70bdbaf24769"
   },
   "source": [
    "Calling embedding model with different instruction pair"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "14a33b32-1adb-4b94-a460-3d6ba1f566af",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "14a33b32-1adb-4b94-a460-3d6ba1f566af",
    "outputId": "e580dace-f17f-41d8-baf2-8a9cd7267d81"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load INSTRUCTOR_Transformer\n",
      "max_seq_length  512\n"
     ]
    }
   ],
   "source": [
    "instructor = (\n",
    "    get_registry()\n",
    "    .get(\"instructor\")\n",
    "    .create(\n",
    "        source_instruction=\"represent the docuement for retreival\",\n",
    "        query_instruction=\"represent the query for retrieval\",\n",
    "    )\n",
    ")\n",
    "\n",
    "\n",
    "class Schema(LanceModel):\n",
    "    vector: Vector(instructor.ndims()) = instructor.VectorField()\n",
    "    text: str = instructor.SourceField()\n",
    "\n",
    "\n",
    "db = lancedb.connect(\"~/.lancedb-qa\")\n",
    "tbl = db.create_table(\"intruct-multitask-qa\", schema=Schema, mode=\"overwrite\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "72ff5950-1c17-4890-9faf-ea15d2ccb585",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "4627e0c9b7c94b97b3ef909e46a7866e",
      "3c45fb44eebc46e6b9469cb246f5a9cc",
      "068a2e8af19d47a5988b8a7005a26662",
      "4e47fb4320ce4c9284c701368611b390",
      "f7eaf13230b042ae902ccd40b82ca730",
      "7c51c71225a04b33ac9c29416ed2a063",
      "3597baa5269b4f488319452d8dc6c1b0",
      "b16c14d65f0f420fba49d190834de508",
      "e7c50595006d4f92ba7b86036372ade7",
      "5a006c198d974dcdb542639ba1ba893f",
      "6fa9c7b3e550426a891ed4cc76335670"
     ]
    },
    "id": "72ff5950-1c17-4890-9faf-ea15d2ccb585",
    "outputId": "a30a1606-376f-405e-ad77-016e49ddeabf"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4627e0c9b7c94b97b3ef909e46a7866e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_qa = [\n",
    "    {\n",
    "        \"text\": \"A canvas painting is artwork created on a canvas surface using various painting techniques and mediums like oil, acrylic, or watercolor. It is popular in traditional and contemporary art, displayed in galleries, museums, and homes.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"A cinema, also known as a movie theater or movie house, is a venue where films are shown to an audience for entertainment. It typically consists of a large screen, seating arrangements, and audio-visual equipment to project and play movies.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"A pocket watch is a small, portable timekeeping device with a clock face and hands, designed to be carried in a pocket or attached to a chain. It is typically made of materials such as metal, gold, or silver and was popular during the 18th and 19th centuries.\"\n",
    "    },\n",
    "    {\n",
    "        \"text\": \"A laptop is a compact and portable computer with a keyboard and screen, ideal for various tasks on the go. It offers versatility for browsing, word processing, multimedia, gaming, and professional work.\"\n",
    "    },\n",
    "]\n",
    "\n",
    "tbl.add(data_qa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e330fd10-c7a2-4fa5-99f9-ca616858f206",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "d5495e0900c74b46adb004bda13c0bc7",
      "170ac432b17543e5a1764af2f63b867d",
      "cea70a622be148849476289b6b25cbe7",
      "3dfcd5f5889540faa7e065b76a61970a",
      "990f04232cbf439bb5fb691e19fa82b2",
      "7599c0e6a7774a6280ce9f2a0cf94407",
      "44455356d2714b63a4332f67d2794d3d",
      "a81b63bb55654633acbab296c2467a7c",
      "0445e8d150fc4e76abd7f13abf5984ae",
      "5a5d98af254f4276915805a88b0b2562",
      "173aaee0be2843e3aadd5daecf4d8ce1"
     ]
    },
    "id": "e330fd10-c7a2-4fa5-99f9-ca616858f206",
    "outputId": "2343d5b5-4e22-469d-d15b-21af658682bf"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d5495e0900c74b46adb004bda13c0bc7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query = \"what is a cinema\"\n",
    "result = tbl.search(query).limit(1).to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "cbc5340a-41a6-4958-a037-e60f249f7012",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cbc5340a-41a6-4958-a037-e60f249f7012",
    "outputId": "57ceeee6-80b9-4fc3-8bad-5fa26c104d21"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              vector  \\\n",
      "0  [0.02184453, 0.0017777232, 0.022723947, 0.0497...   \n",
      "\n",
      "                                                text  _distance  \n",
      "0  A cinema, also known as a movie theater or mov...   0.131036  \n"
     ]
    }
   ],
   "source": [
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11fa2672-8b9b-4a32-a3a6-a5074c2f9a94",
   "metadata": {
    "id": "11fa2672-8b9b-4a32-a3a6-a5074c2f9a94"
   },
   "source": [
    "Thanks, for more such examples, please visit [LanceDB](https://github.com/lancedb/vectordb-recipes/tree/main)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
